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CLAIMS 

What is claimed is: 

1 . A processor-based method for determining difficulty measures for training 
cases used in developing a solution to a problem, comprising: 

1 providing a set of training cases having respectively associated difficulty measures; 

2 operating a candidate solution on a particular training case; 

3 determining a performance measure of the candidate solution operating on the 

4 selected training case; 

5 determining a credibility rating of the candidate solution, the credibility rating 

6 indicating a proficiency level of the candidate solution at solving training cases; and 

7 modifying the difficulty measure of the particular training case based on the 

8 performance measure of the candidate solution operating on particular training case and 

9 the credibility rating of the candidate solution. 

1 2. The method of claim 1, wherein determining the credibility rating 

2 comprises: 

3 selecting one or more training cases from the set of training cases based on the 

4 difficulty measures of the one or more training cases; 

5 determining performance measures of the candidate solution operating on each of 

6 the one or more training cases; and 

7 computing the credibility rating based on the performance measures of the 

8 candidate solution operating on each of the one or more training cases. 

1 3. The method of claim 2, wherein the one or more training cases does not 

2 include the particular training case. 

1 4. The method of claim 1, wherein providing the set of training cases having 

. 2 respectively associated difficulty measures comprises initializing a difficulty measure of 

3 each training case in the set of training cases to a predetermined value. 
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1 5. The method of claim 4, wherein the predetermined value is a maximum 

2 value. 

1 6. The method of claim 1, wherein: 

2 providing the set of training cases comprises associating each training case in the 

3 set of training cases with a target output; 

4 operating the candidate solution on the particular training case comprises obtaining 

5 an output from the candidate solution operating on the particular training case; and 

6 determining the performance measure of the candidate solution operating on the 

7 particular training cases comprises comparing the candidate solution output to a target 

8 output of the particular training case. 

1 7. ; The method of claim 6, wherein comparing the candidate solution output to 

2 the target output of the particular training case comprises calculating a value 

3 corresponding to a deviation between the candidate solution output and the target output of 

4 the particular training case. 

1 8. The method of claim 1, wherein modifying the difficulty measure of the 

2 particular training case comprises modifying the difficulty measure based on a weighted 

3 average of the performance measure and a previous value of the difficulty measure. 

1 9. The method of claim 8, wherein a weight of the weighted average is based 

2 on the credibility rating and a base learning rate. 

1 10. The method of claim 1 ? wherein modifying the difficulty measure 

2 comprises maintaining the difficulty measure within a predetermined interval. 

1 1 1 . A processor-based method for solving a problem, comprising: 

2 providing a set of candidate solutions; 

3 providing a set of training cases having respectively associated difficulty measures; 

4 operating a candidate solution on a particular training case; 

5 determining a performance measure of the candidate solution operating on the 

6 particular training case; 
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7 determining a credibility rating of the candidate solution, the credibility rating 

8 indicating a proficiency level of the candidate solution at solving training cases; 

9 modifying the difficulty measure of the particular training case based on the 

10 performance measure of the candidate solution and the credibility rating of the candidate 

1 1 solution; 

12 evaluating the candidate solution; and 

1 3 modifying the set of candidate solutions based on the evaluation. 

1 12. The method of claim 1 1 , wherein: 

2 providing the set of training cases comprises respectively associating the training 

3 cases in the set of training cases with target outputs; 

4 operating the candidate solution on the particular training case comprises obtaining 

5 an output from the candidate solution operating in the context of the particular training 

6 case; and 

7 determining the performance measure of the candidate solution operating on the 

8 particular training case comprises comparing the candidate solution output to a target 

9 output of the particular training case. 

1 13. The method of claim 1 1, wherein evaluating the candidate solution 

2 comprises: 

3 selecting one or more training cases based on the difficulty measures of the one or 

4 more training cases; and 

5 operating the candidate solution on each of the one or more selected training cases. 

1 14. The method of claim 13, wherein selecting the one or more training cases 

2 comprises biasing selection of the one or more training cases towards more difficult 

3 training cases in the set of training cases. 

1 15. The method of claim 13, wherein selecting each of the one or more training 

2 cases comprises: 

3 randomly selecting a subset of training cases from the set of training cases; and 
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4 selecting a training case from the subset of training cases based on the difficulty 

5 measure of the selected training case. 

1 16. The method of claim 15, wherein selecting the training case comprises 

2 selecting the training case having a relatively high difficulty measure in comparison with 

3 other training cases in the subset of training cases. 

1 17. The method of claim 13, wherein selecting each of the one or more training 

2 cases comprises: 

3 selecting a subset of training cases from the set of training cases based on the 

4 difficulty measures of members of the subset; and 

5 selecting a training case from the subset. 

1 18. The method of claim 13, wherein selecting each of the one or more training 

2 cases comprises selecting a training case based on a probability computed as a function of 

3 a difficulty measure of the selected training case and the difficulty measures of members 

4 of the set of training cases. 

1 19. The method of claim 1 1 , wherein determining the performance measure of 

2 the candidate solution comprises calculating a value corresponding to a deviation between 

3 an output of the candidate solution and the target output. 

1 20. The method of claim 1 1 , wherein modifying the set of candidate solutions 

2 comprises modifying one or more particular candidate solutions in the set of candidate 

3 solutions. 

1 21 . The method of claim 1 1, wherein modifying the set of candidate solutions 

2 comprises forming a new set of candidate solutions. 

1 22. A system, comprising: 

2 an input unit configured to provide a set of training cases and a set of candidate 

3 solutions, each training case having an associated difficulty measure; and 
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4 a processor configured to determine a credibility rating of a candidate solution and 

5 a performance measure of the candidate solution operating on a particular training case, 

6 the processor further configured to modify a difficulty measure of the particular training 

7 . case based on the performance measure and the credibility rating, 

8 wherein the credibility rating indicates a proficiency of the candidate solution at 

9 solving training cases. 

1 23. The system of claim 22, wherein the processor is configured to select one 

2 or more training cases from the set of training cases based on difficulty measures of the 

3 one or more selected training cases, determine performance measures of the candidate 

4 solution operating on each of the one or more selected training cases, and compute the 

5 credibility rating based on the performance measures of the candidate solution operating 

6 on each of the one or more training cases. 

1 24. The system of claim 22, wherein the processor is configured to obtain an 

2 output from the candidate solution operating on the particular training case, compare the 

3 candidate solution output and a target output of the particular training case, and determine 

4 the performance measure of the candidate solution based on the comparison. 

1 25. The system of claim 22, wherein the processor is configured to modify the 

2 difficulty measure of the particular training case based on a weighted average of the 

3 performance measure and a previous value of the difficulty measure. 

1 26. The system of claim 22, wherein the processor is further configured to 

2 evaluate the candidate solution and modify the set of candidate solutions based on the 

3 evaluation. 

1 27. The system of claim 26, wherein the processor is configured to select one 

2 or more training cases based on the difficulty measures of the one or more training cases, 

3 operate the candidate solution on the one or more training cases, determine a performance 

4 measures of the candidate solution operating on the one or more selected training cases, 

5 and evaluate the candidate solution based on the performance measures. 
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1 28. The system of claim 27, wherein the processor is configured to bias 

2 selection of the one or more training cases towards training cases that are more difficult to 

3 solve than other training cases in the set of training cases. 

1 29. A system for determining training case difficulty measures, comprising: 

2 means for providing, a set of training cases having respectively associated difficulty 

3 measures; 

4 means for operating a candidate solution on a particular training case; 

5 means for determining a performance measure of the candidate solution operating 

6 on the selected training case; 

7 means for determining a credibility rating of the candidate solution, the credibility 

8 rating indicating a proficiency of the candidate solution at solving training cases; and 

9 means for modifying the difficulty measure of the particular training case based on 

1 0 the performance measure of the candidate solution operating on particular training case 

1 1 and the credibility rating of the candidate solution. 

1 30. An article of manufacture for determining a solution to a problem, 

2 comprising: 

3 a computer-readable medium configured with instructions for causing a computer 

4 to perform the method of, 

5 providing a set of candidate solutions; 

6 providing a set of training cases having respectively associated difficulty measures; 

7 operating a candidate solution on a particular training case; 

8 determining a performance measure of the candidate solution operating on the 

9 particular training case; 

10 determining a credibility rating of the candidate solution, the credibility rating 

1 1 indicating a proficiency of the candidate solution at solving training cases; 

12 modifying the difficulty measure of the particular training case based on the 

13 performance measure of the candidate solution and the credibility rating of the candidate 

14 solution; 

1 5 evaluating the candidate solution; and 

16 modifying the set of candidate solutions based on the evaluation. 
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